Lecture 7

Working With Non-Binary Outcomes: Spread

Review

Where We Are

	income	education	county
1	$120k	16	kern
2	$4k	13	kern
3	$15k	12	kern
4	$15k	13	kern
⋮
2271	$0k	20	tulare

We’ve talked about estimating the mean of a population.
And means of subpopulations, too. And differences in those.
We’ve been looking the incomes of California residents with and without 4-year degrees.
- Our population is the set of all California residents meeting some age and education criteria.
- And we’ll think of earnings as being drawn with replacement from that population.
To think carefully about the relationship between earnings and the population, we’ll use random variables.

Imagining a Population

	income	education	county
1	$51k	14	sacramento
2	$0k	16	monterey
3	$0k	13	unknown
⋮
14194	$62k	12	LA

	income	education	county
1	$120k	16	kern
2	$4k	13	kern
3	$15k	12	kern
⋮
2271	$0k	20	tulare

We don’t have much information about the people in our population. Except the ones in earnings.
But, for the sake of visualization, I’ve made some up.
- We’ll put visualizations of this fake population on the left.
- Visualizations of the (real) sample will be on the right.

Unbiasedness

We’ve shown that using sample means as estimators of population means gives us unbiased estimators.
- That is, estimators whose expected values are the thing we want to estimate.
- Intuitively, that means their sampling distributions are where we want them to be.
- Like the plot on the left. Not the one on the right.
This is essentially necessary to get correctly calibrated interval estimates.
It’s more or less sufficient too.
- Calibration via bootstrap works for every unbiased estimator we’ll talk about this semester.
- Things are more complicated for fancy machine learning stuff, but that’s beyond the scope.

We Can Trust This Interval Estimate

\[ \mu(1)-\mu(0) \in 41.96K \pm 5.77K \qqtext{is a 95\% confidence interval} \]

This is a 95% confidence interval for the difference in mean income between California residents with and without 4-year degrees.
In 95% of surveys, it’ll contain the different in population means. That’s the point of all of this.

The NSW version

\[ \mu(1)-\mu(0) \in 1.79K \pm 1.34K \qqtext{is a 95\% confidence interval} \]

This is a 95% confidence interval for the difference in mean income between California residents with and without 4-year degrees.
In 95% of surveys, it’ll contain the different in population means. That’s the point of all of this.

The NSW version

\[ \mu(1)-\mu(0) \in 1.79K \pm 1.34K \qqtext{is a 95\% confidence interval} \]

This is a 95% confidence interval for the difference in mean income between treated and untreated people in the National Supported Work experiment.
That’s in 1978 dollars. Accounting for inflation, it’s about $3.5$ times that in 2022 dollars.

\[ \mu(1)-\mu(0) \in 6.28K \pm 4.68K \qqtext{is a 95\% confidence interval (in 2022 dollars)} \]

What’s The Difference?

California 4-year degree vs no 4-year degree

NSW treated vs untreated (adjusted for inflation)

Do these intervals tell the same story about the income differences between the groups involved?

In my opinion, no.
The California interval tells us there’s a big difference between these groups in the population.
- It’s 36.20K to 47.73K. That’s a big difference no matter which end of the interval the population difference is on.
The NSW interval tells us we don’t know.
- It’s 1.60K to 10.96K in 2022 dollars.
- That’s a smallish difference at one end of the interval and a big one at the other.
What we want is to be able to anticipate this and design our studies to avoid it.
The key to this is understanding the spread of the distribution of our estimator. That’s what we’ll talk about today.

Probability Stuff

Review: Observations as Random Variables

The notation $y_j$ refers to a number.
- It’s the income of the $j$th person in our population. That’s just some person.
The notation $Y_i$ refers to something else.
- It’s the income of the $i$th person we call in our survey. That’s not a person.
- That’s the result of a random process—the roll of a die.
To summarize this result, we talk about the probability distribution of $Y_i$.

Notation Conventions

Random variables are written in uppercase: $X$, $Y$, $Z$, etc. Constants are written in lowercase: $x$, $y$, $z$, etc.
We’ll use the same letter for a random variable and the value it takes on. $x$ is a possible value of $X$, $y$ of $Y$, etc.
Estimators are also random variables, but instead of uppercase we use a hat: $\hat \mu$, $\hat \sigma$, etc.
We’ll use the same letter for the estimator and what it’s meant to estimate. $\hat \mu$ is an estimate of $\mu$.

Review: Expectations

	income	education	county
1	$51k	14	sacramento
2	$0k	16	monterey
3	$0k	13	unknown
⋮
14194	$62k	12	LA

	roll	income50k	income	education	county
1	1017	1	$120k	16	kern
2	8004	0	$4k	13	kern
3	4775	0	$15k	12	kern
⋮
2271	7117	0	$0k	20	tulare

Often the thing we care about is the expected value of $Y_i$. Or some related quantity.
- The expected value of $Y_i$ is the probability-weighted average of the values it can take on.
- What’s nice about this is that we can think of this in ‘uncollapsed’ terms.
When we sample as usual, this is just the population mean.
The collapsed form is, in a sense, just summing in a different order.

\[ \begin{aligned} \mathop{\mathrm{E}}[Y_i] = \sum_y P(Y_i = y) \times y = \sum_y \qty(\sum_{j: y_j = y} \frac{1}{m}) \times y = \frac{1}{m}\sum_y\sum_{j:y_j = y} y = \frac{1}{m}\sum_{j=1}^m y_j \end{aligned} \]

Rules for Expectations

The law of iterated expectations

\[ E \{ E( Y \mid X ) \} \quad \text{ for any random variables $X, Y$} \]

To calculate the mean of $Y$, we can average within subpopulations, then across subpopulations.

Irrelevance of independent conditioning variables

\[ E( Y \mid X, X' ) = E( Y \mid X ) \quad \text{ when $X'$ is independent of $X,Y$ } \]

If $X'$ is unrelated to $X$ and $Y$, holding it constant doesn’t affect the relationship between them.

Linearity of Expectations

\[ E ( a Y + b Z ) = aE(Y) + bE(Z) \quad \text{ for random variables $Y, Z$ and numbers $a,b$ } \]

We can distribute expectations through sums and multiplication by constants.

Linearity of Conditional Expectations

\[ E\{ a(X) Y + b(X) Z \mid X \} = a(X)E(Y \mid X) + b(X)E(Z \mid X) \]

When we condition on $X$, we can distribute as if functions of $X$ were constants.

An Important Consequence.

The expectation of a product of independent random variables is the product of their expectations. \[ \mathop{\mathrm{E}}[YZ] = \mathop{\mathrm{E}}[Y]\mathop{\mathrm{E}}[Z] \qqtext{when $Y$ and $Z$ are independent} \]

This comes up a lot in variance calculations. Let’s prove it. It’ll be good practice.

\[ \begin{aligned} \mathop{\mathrm{E}}[YZ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{YZ \mid Z}] && \text{iterated expectations} \\ &= \mathop{\mathrm{E}}\qty[ Z\mathop{\mathrm{E}}\qty{Y \mid Z} ] && \text{linearity of conditional expectations} \\ &= \mathop{\mathrm{E}}\qty[ Z \mathop{\mathrm{E}}\qty{Y} ] && \text{irrelevance of independent conditioning variables} \\ &= \mathop{\mathrm{E}}\qty{Y}\mathop{\mathrm{E}}[Z] && \text{linearity of expectations} \end{aligned} \]

Spread

To summarize the spread of a distribution, we often talk about variance.
The variance is an expectation: it’s the expected squared difference between a random variable and its expectation. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y) = \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] \]
To put that back on the same scale as $Y$—instead of the scale of $Y^2$—we take it’s square root. That’s the standard deviation.

\[ \text{standard deviation}(Y) = \sqrt{\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y)} \]

We can talk about spread outright (left) — that’s (unconditional) variance.
Or spread within groups (right) — that’s conditional variance.

Conditional Variance

The conditional variance describes the spread of one random variable within groups.
Like the conditional expectation of the random variable, there are two related concepts.

The conditional variance function. This is a function of $x$. \[ \sigma^2(x) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X=x) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X=x] \]
The conditional variance $\sigma^2(X) = \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X)$. This is a random variable. \[ \sigma^2(X) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X] \]

The Conditional Variance Function

\[ \sigma^2(x) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X=x) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X=x] \]

This is the variance of $Y$ within the subpopulation of people with $X=x$.
- $\sigma(1)$ is a number describing the spread of the green dots in our population.
- $\sigma(0)$ is a number describing the spread of the red dots in our population.
It’s the conditional expectation function of the random variable $\textcolor{blue}{\{Y-\mathop{\mathrm{E}}(Y \mid X)\}^2}$.
We call it ’the conditional variance function of $Y$ given $X=x$.

The Conditional Variance

\[ \sigma^2(X) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X] \]

This is the variance of $Y$ within a random subpopulation of people.
- It’s the conditional variance function evaluated at the random variable $X$.
- And the conditional expectation of the random variable $\textcolor{blue}{\{Y-\mathop{\mathrm{E}}(Y \mid X)\}^2}$.
We call it ‘the conditional variance of $Y$ given $X$’.
This is the sort of thing that shows up when you use the law of iterated expectations.

The Law of Total Variance

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \mathop{\mathrm{E}}\qty{\mathop{\mathrm{\mathop{\mathrm{V}}}}( Y \mid X ) } + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{\mathop{\mathrm{E}}( Y \mid X ) } \]

The Law of Total Variance relates Conditional and Unconditional Variance
It’s like the Law of Total Expectation, but for Variance
It’s a useful way to decompose the variance a random variable.
Think about where most of the variance in $Y$ is coming from in the two populations shown below.

An Activity

Working with your neighbor, think about why we call it this.
Try to put the three terms into words.
- $\mathop{\mathrm{\mathop{\mathrm{V}}}}[Y]$ is the …
- $\mathop{\mathrm{E}}\qty{ \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) }$ is the …
- $\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{\mathop{\mathrm{E}}( Y \mid X)}$ is the …
Do this in two ways.
1. Using ‘Random Variables Language’,
  - e.g. ‘the expected value of the conditional variance of Y given X’
2. Using ‘Population Language’
  - e.g. ‘the variance of income (Y) within the random subpopulation with degree status X’

Proving the Law of Total Variance

Starting Point: Interpreting Variance as Excess

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y) &= \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] && \text{ the \emph{spread} idea: the mean square of a \emph{centered version} of $Y$ } \\ &= \mathop{\mathrm{E}}\qty(Y^2) - \qty{\mathop{\mathrm{E}}\qty(Y)}^2 && \text{ the \emph{excess} idea: the average amount $Y^2$ \emph{exceeds} the square of its mean}. \end{aligned} } \]

Let’s think, in terms of the plot, about why there is excess in the square.

Suppose $Y$ takes on three values — the dots —with equal probability.
- Its expectation is the average height of the black dots.
- And its expectation, squared, is the square of this.
Its square $Y^2$ takes on the the square of those values—the squares — with equal probability.
- Its expectation is the average height of the squares.
Why is the second one bigger?
- Because squaring increases big draws of Y more than it decreases small ones.
- Visually, the line from $Y$ to $Y^2$ is longer when $Y$ is bigger than when it’s smaller.
We’ll mostly think about variance in terms of spread, but the ‘excess’ formula comes up in calculations.

Equivalence Proof

Claim. The spread and excess formulas for variance are equivalent. \[ \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] =\mathop{\mathrm{E}}[ Y^2] - \qty{\mathop{\mathrm{E}}(Y)}^2 \]

\[ \small{ \begin{aligned} \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] &= \mathop{\mathrm{E}}\qty[ Y^2 - 2 Y \mathop{\mathrm{E}}(Y) + \qty{\mathop{\mathrm{E}}(Y)}^2 ] && \text{ FOIL } \\ &= \mathop{\mathrm{E}}[ Y^2] - \mathop{\mathrm{E}}\qty[2 Y \mathop{\mathrm{E}}(Y)] + \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}(Y)}^2] && \text{Distributing expectations (linearity)} \\ &= \mathop{\mathrm{E}}[ Y^2] - 2\mathop{\mathrm{E}}(Y)\mathop{\mathrm{E}}\qty[Y] + \qty{\mathop{\mathrm{E}}(Y)}^2\mathop{\mathrm{E}}[1] && \text{Pulling constants out of expectations (linearity)} \\ &= \mathop{\mathrm{E}}[ Y^2] - 2\qty{\mathop{\mathrm{E}}(Y)}^2 + \qty{\mathop{\mathrm{E}}(Y)}^2] && \text{Recognizing a product of something with itself as a square} \\ &= \mathop{\mathrm{E}}[ Y^2] - \cancel{2}\qty{\mathop{\mathrm{E}}(Y)}^2 + \cancel{\qty{\mathop{\mathrm{E}}(Y)}^2}] && \text{Subtracting} \end{aligned} } \]

Conditional Variance as Excess

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) &= \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y\mid X)}^2 \mid X ] && \text{ the \emph{spread} idea: the mean square of a conditionally centered version of $Y$ } \\ &= \mathop{\mathrm{E}}\qty(Y^2 \mid X) - \qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2 && \text{ the \emph{excess} idea: the average amount $Y^2$ exceeds the square of its cond. mean} \end{aligned} } \]

We can think of the Conditional Variance as excess, too.
It’s the same as the unconditional case, but within subpopulations.

Proof

Claim. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X)]} + \textcolor{green}{\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}(Y \mid X)]} \]

Rewrite the variances, conditional and unconditional, in excess form.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X) - \qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]} + \textcolor{green}{\mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}(Y \mid X)^2} - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

Distribute the outer expectations in the first term.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X)] - \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]} + \textcolor{green}{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ] - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

Cancel matching terms

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X)] \cancel{- \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]}} + \textcolor{green}{\cancel{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ]} - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

Use the law of iterated expectations to simplify the remaining terms.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{E[Y^2] \cancel{- \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]}} + \textcolor{green}{\cancel{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ]} - (\mathop{\mathrm{E}}\qty{Y})^2} \]

Variance Calculations

The Variance of The Mean

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i Y_i - \mathop{\mathrm{E}}\qty( \frac{1}{n}\sum_i Y_i ) }^2 ] && \text{definition}\\ &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i (Y_i - \mathop{\mathrm{E}}Y_i) }^2 ] && \text{Linearity} \\ &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i Z_i }^2 ] && \text{for} \ \ Z_i = Y_i - \mathop{\mathrm{E}}Y_i \\ &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] && \text{via FOIL} \end{aligned} \]

FOIL?

Remember FOIL from high school algebra?
When we square a sum, we get all products of terms in the sum.

\[ (a+b)^2 = \textcolor[RGB]{0,0,255}{(a+b)}\textcolor[RGB]{248,118,109}{(a+b)} = \textcolor[RGB]{0,0,255}{a}\textcolor[RGB]{248,118,109}{a} + \textcolor[RGB]{0,0,255}{a}\textcolor[RGB]{248,118,109}{b} + \textcolor[RGB]{0,0,255}{b}\textcolor[RGB]{248,118,109}{a} + \textcolor[RGB]{0,0,255}{b}\textcolor[RGB]{248,118,109}{b} = a^2 + 2ab + b^2 \]

Same deal for bigger sums.

\[ \qty{ \sum_i Z_i }^2 = \textcolor[RGB]{0,0,255}{\sum_i Z_i}\textcolor[RGB]{248,118,109}{\sum_j Z_j} = \sum_i \sum_j \textcolor[RGB]{0,0,255}{Z_i}\textcolor[RGB]{248,118,109}{Z_j} \]

Completing the Calculation

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] \quad \text{ for } \quad Z_i = Y_i - \mathop{\mathrm{E}}Y_i \quad && \text{via FOIL} \end{aligned} \]

Activity. Complete the calculation to get a simple expression.

Hints.

Use linearity to push the expectation inside the sum.
Think about what $\mathop{\mathrm{E}}Z_i Z_j$ is when $i=j$ and $i\neq j$.
Recall that when $i \neq j$ they’re independent.
What is $\mathop{\mathrm{E}}Z_i^2$?

Rules. \[ \small{ \begin{aligned} \text{1.} \quad & \mathop{\mathrm{E}}\qty[ \sum_i a_i Y_i] = \sum_i a_i \mathop{\mathrm{E}}Y_i \quad \text{ for RVs $Y_i$ and constants $a_i$.} \\ \text{2.} \quad & \mathop{\mathrm{E}}\qty[ Z_i Z_j] = \mathop{\mathrm{E}}[Z_i] \mathop{\mathrm{E}}[ Z_j] \quad \text{if $Z_i$ and $Z_j$ are independent.} \\ \end{aligned} } \]

The Variance of The Mean: Answered

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] && \text{via FOIL} \\ &= \frac{1}{n^2} \sum_i \sum_j \mathop{\mathrm{E}}Z_i Z_j && \text{via Linearity } \\ &= \frac{1}{n^2} \sum_i\sum_j \begin{cases} \mathop{\mathrm{E}}Z_i^2 & \text{if } i=j \\ \mathop{\mathrm{E}}Z_i \mathop{\mathrm{E}}Z_j & \text{if } i\neq j \end{cases} && \text{using the independence of $Z_i$ and $Z_j$} \\ &= \frac{1}{n^2} \sum_i\sum_j \begin{cases} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i) & \text{if } i=j \\ 0 \times 0 & \text{if } i\neq j \end{cases} && \text{since $Z_i=Y_i-\mathop{\mathrm{E}}Y_i$} \\ &= \frac{1}{n^2} \times n \times \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i) = \frac{\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i)}{n} = \frac{\sigma^2}{n}. && \text{counting the copies of $\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i)$ in the sum} \\ \end{aligned} \]

The Variance of the Subsample Mean

Claim. The variance of our point estimate is the expected value of the subpopulation variance divided by the number of people in the subsample. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1)] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{N_1} ] \quad \text{ for } \quad N_1 = \sum_{i=1}^n X_i \]

This is like what we saw in the unconditional case, with a twist.
The number of people in the subsample is random, so instead of $1/n$, we have $\mathop{\mathrm{E}}[ 1/N_1 ]$.
Let’s prove it.
- It’s not hard, but it’s a bit long. The key is organization.
- We’ll break things down into manageable pieces and focus on one piece at a time.

Step 1. Total Variance

We’ll start by using the Law of Total Variance, conditioning on $X_1 \ldots X_n$.
And then we’ll have to calculate two terms.
1. The expected value of the conditional variance
2. The variance of the conditional expectation \[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n }] + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n} ] \\ \end{aligned} \]
Plugging in the definitions of variance and conditional variance, these terms are

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n} &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} ]^2 \mid X_1 \ldots X_n) \\ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}] &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} - \mathop{\mathrm{E}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}]}^2 ) \\ \end{aligned} \]

We see that the conditional expectation of our estimator comes up in both terms.
I’ve claimed it’s just $\mu(1)$. Let’s take a look at that.

Step 2. The Estimator’s Conditional Mean

To calculate the conditional mean of our estimator, we’ll revisit an argument we’ve seen before.
We’ll look at our argument that our argument that the subsample mean $\hat\mu(1)$ is unbiased.
- We’ll see that it’s conditionally unbiased.
- That is, its expected value is $\mu(1)$, the thing we want to estimate
- conditional on the values of the covariates $X_1 \ldots X_n$.

\[ \begin{aligned} \mathop{\mathrm{E}}\qty[\hat\mu(1) \mid X_1 \ldots X_n] &=\mathop{\mathrm{E}}\qty[\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} \mid X_1 \ldots X_n] \\ &=\frac{\sum_{i=1}^{n} X_i \mathop{\mathrm{E}}\qty{ Y_{i} \mid X_1 \ldots X_n}}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i\mathop{\mathrm{E}}\qty{ Y_{i} \mid X_i}}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i \mu(X_i)}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i\mu(1)}{\sum_{i=1}^{n}X_{i}} \\ &=\mu(1) \times \frac{\sum_{i=1}^{n}X_i}{\sum_{i=1}^{n}X_{i}} = \mu(1) \end{aligned} \]

Step 3. Simplifying Our Total Variance Decomposition

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n }] + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n} ] \end{aligned} \]

where

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n} &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} ]^2 \mid X_1 \ldots X_n) \\ &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n) \\ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}] &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} - \mathop{\mathrm{E}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}]}^2 ) \\ &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mu(1)} - \mathop{\mathrm{E}}\qty[\mu(1)] }^2 ) = 0\\ \end{aligned} \]

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty[\hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n }] \]

Step 4. Rewriting as a Sum of Mean-Zero Random Variables

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty[\hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n }] \]

Now we’ll substitute in definitions and simplify.
We’ll end having to work with the conditional mean of a squared sum.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\hat \mu(1)-\mu(1))^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} - \mu(1))^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} - \frac{\mu(1)\sum_{i=1}^n X_i}{\sum_{i=1}^n X_i })^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)}}{\sum_{i=1}^{n}X_{i}})^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)})^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \end{aligned} } \]

Step 5

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)})^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \]

To work with this squared sum, we do our FOIL-like expansion. This is …

\[ \small{ \begin{aligned} \qty(\sum_{i}Z_i)^2 = \qty(\sum_{i}Z_i) \qty(\sum_{j}Z_i) = \sum_i\sum_j Z_i Z_j \quad && \text{ for } \quad Z_i = X_i \qty{Y_{i}-\mu(1)} \\ \text{or equivalently, using the \textbf{indicator trick}} && \text{ for } \quad Z_i = X_i\qty{Y_i - \mu(X_i)} \end{aligned} } \]

This lets us push the conditional mean all the way into the sum, so it meets the individual terms.
And when we’ve done that, we can use the irrelevance of independent conditioning variables to simplify.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}Z_i )^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \sum_{i=1}^{n}\sum_{j=1}^n Z_i Z_j \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ \end{aligned} } \]

Step 6

Let’s take a look at one of the terms in the numerator, $\mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}$.
- We’re going to show that the cross-terms — the ones for $i \neq j$ — are zero.
- This will let us drop them from the sum.
- We’ll use the fact that $Z_i$ and $Z_j$ are independent for $i \neq j$.
Let’s rewrite a cross-term using our usual stuff
- the law of iterated expectations, conditioning on $Z_i$
- linearity of expectations;
- irrelevance of independent conditioning variables.

\[\small{ \begin{aligned} \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j} &= \mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}(Z_i Z_j \mid Z_i,X_i,X_j) } \\ &= \mathop{\mathrm{E}}\qty{ Z_i \mathop{\mathrm{E}}(Z_j \mid X_i,X_j) } \\ &= \mathop{\mathrm{E}}\qty{ Z_i \textcolor{blue}{\mathop{\mathrm{E}}(Z_j \mid X_j)} } \quad \text{ where } \\ \textcolor{blue}{\mathop{\mathrm{E}}(Z_j \mid X_j)} &= \mathop{\mathrm{E}}\qty[X_j \qty{Y_j - \mu(X_j)}] \\ &= \mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}\qty[X_j \qty{Y_j - \mu(X_j)} \mid X_j] } \\ &= \mathop{\mathrm{E}}\qty[X_j \qty{ \mathop{\mathrm{E}}\qty[Y_j \mid X_j] - \mu(X_j)}] \\ &= \mathop{\mathrm{E}}\qty[X_j \cross{\qty{ \mu(X_j) - \mu(X_j)}}] = 0. \\ \end{aligned} } \]

Step 7

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \quad \text{ for } \quad Z_i=Y_i-\mu(X_i) \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} \mathop{\mathrm{E}}\qty{Z_i^2 \mid X_i}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \quad \text{because cross-terms are zero}. \end{aligned} \]

All we’ve got to do now is recognize what we have.
Let’s look at a term in the numerator.
- We’ll use iterated expecations and linearity.
- Then the indicator trick.

\[ \small{ \begin{aligned} \mathop{\mathrm{E}}\qty[Z_i^2 \mid X_i] &= \mathop{\mathrm{E}}\qty[ X_i\qty{Y_i - \mu(X_i)}^2 \mid X_i] = X_i\mathop{\mathrm{E}}\qty[\qty{Y_i - \mu(X_i)}^2 \mid X_i] = X_i\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[Y_i \mid X_i] = X_i \sigma^2(X_i) = X_i\sigma^2(1) \end{aligned} } \]

We finish up by substituting this back into what we started with.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} \mathop{\mathrm{E}}\qty{Z_i^2 \mid X_i}}{\qty(\sum_{i=1}^{n}X_{i})^2}] = \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} X_i \sigma^2(1)}{\qty(\sum_{i=1}^{n}X_{i})^2}] = \mathop{\mathrm{E}}\qty[ \frac{\cancel{\sum_{i=1}^{n} X_i }\sigma^2(1)}{\qty(\sum_{i=1}^{n}X_{i})^{\cancel{2}}}] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{\sum_{i=1}^{n}X_{i}}] = \sigma^2(1) \times \mathop{\mathrm{E}}\qty[ \frac{1}{N_1}]. \end{aligned} } \]

The Variance of Differences in Means

Claim. The variance of the difference in subsample means is the sum of the variances of the subsample means themselves. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0)] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{N_1} + \frac{\sigma^2(0)}{N_0}] \quad \text{ for } \quad N_x = \sum_{i=1}^n 1_{x}(X_i) = \begin{cases} 1 &\text{if} X_i=x \\ 0 &\text{otherwise}\end{cases} \]

Prove it. This is about understanding the logic of the proof we just did. Start with the law of total variance.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0)] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n] ] + \mathop{\mathrm{\mathop{\mathrm{V}}}}[\mathop{\mathrm{E}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n] ] \]

Here are some things to think about.

What is the conditional expectation of the estimator given $X_1 \ldots X_n$, i.e. $\mathop{\mathrm{E}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n]$? And what is its variance?
$\mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n]$ is the conditional mean $\mathop{\mathrm{E}}[Z^2 \mid X_i]$ of some random variable $Z$.
What is that random variable?
Can we write it in a way so that, when we square it, the cross terms are zero?
Hint: $X_i(1-X_i)=0$.